System and method for geographically organizing and classifying businesses on the world-wide web

ABSTRACT

A method and search engine for classifying a source publishing a document on a portion of a network, includes steps of electronically receiving a document, based on the document, determining a source which published the document, and assigning a code to the document based on whether data associated with the document published by the source matches with data contained in a database. An intelligent geographic- and business topic-specific resource discovery system facilitates local commerce on the World-Wide Web and also reduces search time by accurately isolating information for end-users. Distinguishing and classifying business pages on the Web by business categories using Standard Industrial Classification (SIC) codes is achieved through an automatic iterative process.

BACKGROUND OF THE INVENTION

The present invention generally relates to a resource discovery systemand method for facilitating local commerce on the World-Wide Web and forreducing search time by accurately isolating information for end-users.For example, distinguishing and classifying business pages on the Web bybusiness categories using the Standard Industrial Classification (SIC)codes is achieved through an automatic iterative process whicheffectively localizes the Web.

DESCRIPTION OF THE RELATED ART

Resource discovery systems have been widely studied and deployed tocollect and index textual content contained on the World-Wide Web.However, as the volume of accessible information continues to grow, itbecomes increasingly difficult to index and locate relevant information.Moreover, global flat file indexes become less useful as the informationspace grows causing user queries to match too much information.

Leading organizations are attempting to classify and organize all of Webspace in some manner. The most notable example is Yahoo, Inc. whichmanually categorizes Web sites under fourteen broad headings and 20,000different sub-headings. Still others are using advanced informationretrieval and mathematical techniques to automatically bring order outof chaos on the Web.

Solutions to solve this information overload problem have been addressedby C. Mic Bowman et al. using Harvest: A Scalable, Customizable ResourceDiscovery and Access System. Harvest supports resource discovery throughtopic-specific content indexing made possible by a very efficientdistributed information gathering architecture. However, these topicspecific brokers require manual construction and they are geared morefor academic and scientific research than commercial applications.

Cornell's SMART engine developed by Gerard Salton uses a thesaurus toautomatically expand a user's search and capture more documents.Individual, Inc. uses this system to sift through vast amounts oftextual data from news sources by filtering, capturing, and rankingarticles and documents based on news industry classification.

The latest attempts for automated topic-specific indexing include theExcite, Inc. search engine which uses statistical techniques to build aself-organizing classification scheme. Excite Inc.'s implementation isbased on a modification of the popular inverted word indexing techniquewhich takes into account concepts (i.e., synonymy and homonymy) andanalyzes words that frequently occur together. Oracle has developed asystem called ConText to automatically classify documents under anine-level hierarchy that identifies a quarter-million differentconcepts by understanding the written English language. ConText analyzesa document and then decides which of the concepts best describe thedocument's topic.

The systems described above all attempt to organize the vast amounts ofdata residing on the Web. However, these mathematical informationretrieval techniques for classifying documents only work when themessage of a document is directly correlated to the words it contains.Attempts to isolate documents by regions or to separate business contentfrom personal content in an automated fashion is not addressed by anyconventional system or structure.

SUMMARY OF THE INVENTION

It is therefore an object of the present invention to provide a methodand system for overcoming the above-mentioned problems of theconventional methods and techniques.

The invention is based on a heuristic algorithm which exploits commonWeb page design principles The key challenge is to ascertain the ownerof a Web page through an iterative process. Knowing the owner of a Webpage helps identify the nature of the content business or personalwhich, in turn, helps identify the geographic location.

In a first aspect of the invention, a method of classifying a sourcepublishing a document on a portion of a network, includes steps ofelectronically receiving a document, based on the document, determininga source which published the document, and assigning a code to thedocument based on whether data associated with the document published bythe source matches with data contained in a database.

In a second aspect, a search engine is provided for use on a network fordistinguishing between business web pages and personal web pages. Thesearch engine includes a mechanism for parsing the content of ahyper-text markup language (HTML) at a web address and searching forcriteria contained therein, a mechanism for analyzing a uniformresources locator (URL) of the web address to determine characteristicsthereof of a web page at the web address, a mechanism for determiningwhether the criteria match with data contained in a database, and amechanism for cross-referencing a match, determined by the determiningmechanism, to a second database, to classify a source which publishedthe web page.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, aspects and advantages will be betterunderstood from the following detailed description of a preferredembodiment of the invention with reference to the drawings, in which:

FIG. 1 shows the process flow diagram of a geographically bound resourcediscovery system including three main components of the invention(sometime referred to below as “MetroSearch”) identified as MetroBot,IPLink, and YPLink;

FIG. 2 depicts the IPLink flow chart, the process for identifying ISPsand Client Directory Paths;

FIGS. 3A-3C are sub-processes of the IPLink flow chart shown in FIG. 2;

FIG. 4 depicts the flow chart of YPLink for identifying business pages;

FIG. 5 is a flow diagram for determining if a given uniform resourceslocator (URL) is a Root URL or a Leaf URL; and

FIG. 6 is a template of a typical business home page.

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT OF THE INVENTION

Referring now to the drawings, and more particularly to FIG. 1, there isshown the general arrangement of a preferred embodiment according to thepresent invention.

The underlying insight behind the invention is that individuals andorganizations responsible for the design, creation, and maintenance oftheir home page generally follow some basic unwritten rules. These rulescan be exploited to automatically identify the owner of the home pagewith a high probability of success. Once the owner of the home page isdetermined, an SIC code is assigned to it by looking up the owner in aYellow Pages database. If a matching entry exists, then the owner is abusiness, otherwise the owner is deemed to be an individual with apersonal home page.

FIG. 1 shows a preferred architecture for implementing a geographicallybound resource discovery system. The main components of interest areMetroBot 126, IPLink 113, and YPLink 112.

The World-Wide Web (“the Web”) 124 is based on a client-serverarchitecture. The Web is the graphical, multimedia portion of theInternet 120. The client side program is a Web browser 100 and theserver side is a computer running the HTTPD program 102. The Web serveris accessed through the Internet by specifying a Uniform ResourceLocator (URL). User-entered queries are sent to a back-end processor orsearch engine 104 which gathers results from various databases 106, 108,110, and 128, and formats the request and presents them back to theuser.

MetroBot 126 is an indexer robot which traverses hyperlinks in HTMLdocuments and indexes the content into a searchable Web index database128. These hyperlinks or URLs point to other Web pages making itpossible to recursively traverse large portions of the Web from asingle, well-chosen URL (seed URL). MetroBot begins its traversal fromknown Root URL 119 such as the home page of a local service provider(SP), such as an internet service provider (ISP). New links that arediscovered are stored in New URLs database 118. These links areprocessed by IPLink 113 and YPLink 112 to extract new Root URLs at whichpoint the whole process repeats itself. Furthermore, YPLink periodicallysupplements its New URL list by querying global search engines 121 usingstrategic keywords (e.g., regional city, county, state names, zip codes,and industry specific terms).

The first level of localization is achieved by limiting URLs toregistered domain names 106. IPLink extracts domain names from the NewURL database and then queries the InterNIC database 122 where records ofregistered domain names containing company name, contact, streetaddress, and Internet Protocol (IP) addresses are kept. This InterNICdatabase can be accessed through the Unix whois(1) command. YPLinkmerges the InterNIC address database 108 with the Yellow Pages data 110.This process is described in detail below.

The next level of localization is more complex since most businesses donot have their own registered domain name. Instead, they have their homepage hosted on local SPs (or ISPs) or Online Service Providers (OSPs)Web Servers.

The first step in solving this problem is for IPLink 113 to characterizeURLs by their IP addresses. Figured 2 and 3A-3C shows the IPLink flowlogic. IPLink identifies the following attributes based on the IPaddresses of New URLs:

True/Virtual Web Servers vs. Shared Web Servers.

ISP vs. Non-ISP hosts.

Root Domain of URLs.

Root Path of URLs.

Client Directory Paths if host is an ISP.

A new URL is retrieved from the New URL database 200 and is parsed intothe domain name and directory path portions. If it is a new domain 205,then its Web IP address (i.e., www.domain.name) is retrieved using theInternet Domain Name Service 122. The Unix nslookup(1) utility 210returns an IP address given a domain name. The corresponding IP addressis stored in the ISP database 114. A reverse lookup 210 of the Web IPaddress is also performed to determine 215 if the given URL is hosted ona true (or virtual) Web server 220 or a shared Web server 225. A domainname with its own unique Web IP address indicates a true or virtual Webserver (non-ISP host). Multiple domain names for a single Web IP addressindicates a shared Web server (ISP host).

The official domain name (Root Domain) 220 and 225 for the IP address isthe domain name of the ISP (master/slave name server informationreturned by whois(1) can also be used to accurately identify the ISP ifthe Root Domain does not correspond to the ISP). Root Domain is onlyused for displaying URL information on search results not for furtherprocessing.

Turning to FIG. 3A, for shared servers 225, the Root Path is determinedby searching 300 for the given domain name in the New URL database 118and finding common directory paths 305. If no match is found 315, theURL will automatically be processed at a later iteration 230, otherwisethe Root Path is set to the matching path 310.

Turning to FIG. 3B, for virtual servers 220, the Root Path is simply theroot directory (‘/’). These servers may or may not be ISPs. If multipledomain names exist for the given IP address 320, then it is classifiedas an ISP 325, otherwise it is processed at a later iteration 330, 235and 240. It is possible for organizations to become ISPs in the futureby simply adding/hosting new domain names on their existing Web servers.

The directory path where the ISP stores its customers Web pages iscalled the ISP Client Directory Path 116. This data is initially createdmanually for a few local ISPs (seed ISPs). This path is identifiedautomatically 335 by searching for the given domain name in the Root URLdatabase 119 and finding common directory paths 340, as shown in FIG.3C. If no match is found 350, then it is processed at a later iteration245. Matching paths 345 point to the ISPs Client Directory Path. Thisprocess improves over subsequent iterations when enough data is gatheredand patterns can be recognized from a large set of ISP Web Servers.

IPLink encompasses the first phase of identifying and characterizing IPaddresses. The next phase is to automatically identify businesses hostedon ISP Web servers.

FIG. 4 shows the YPLink flow chart. YPLink determines if a Web pagebelongs to a business or an individual. YPLink takes its input, a URL,from IPLink. FIG. 4 shows the flow diagram for the YPLink process. Thefirst step after retrieving a URL 400 is determining if it is a “RootURL” or a “Leaf URL” 405.

A Root URL is the entry point for an organization's or individual's homepage on the World-Wide Web. A Root URL may or may not be the same as theHome page. Leaf URLs, on the other hand, are links below anorganization's Root URL. Four factors are considered in determining aRoot URL:

-   -   1. Is the URL hosted on a Service Provider's Web Server?    -   2. Is the URL on a virtual Web Server?    -   3. Does the URL contain a directory path?    -   4. Is the directory path a known Service Provider's Client        Directory?

IPLink determines the SP Client Directory Path as described above. TheISP database 114 contains information about Client Directories forvarious ISPs.

FIG. 5 shows the Root URL flow logic. A given URL is retrieved 500 andparsed into two components: domain name and directory path. The domainname is analyzed to see if it is an ISP 502. If multiple IP addressesare associated with the domain name, then the domain name is an ISP. Ifthe domain name is not an ISP, then the directory path component ischecked 504. A missing directory path signifies a Root URL 506,otherwise it is a Leaf URL 508.

If the domain name is an ISP 510, then it is also a Root URL if nodirectory path exists 512. If a directory path exists 514, then the pathis compared to a list of known ISP Client Directory paths. No match 516indicates a Leaf URL, otherwise the directory path level is analyzed 518for final Root URL determination. If the path is one directory levelbelow the Client Directory path then it is a Root URL 522, otherwise itis a Leaf URL 520.

After a URL is determined to be Root URL, then the home page it pointsto is analyzed 415 to see if it follows some basic guidelines. A typicalhome page layout is illustrated in FIG. 6. Other than following HTMLrequirements, there is no rule or standards for the layout of textualcontent. The key pieces of information required to ascertain the ownerof a Web page are 1) company name, 2) zip code, and 3) telephone number.These three pieces of information do not have to exist in the Root URL.They can reside anywhere among various Leaf URLs beneath a Root URL. Inmany cases, this information is stored in a file called about.html.However, the same information could be stored in other, similarly namedfiles, as would be known to those skilled in the art taking the presentspecification as a whole. The process described below extracts thisinformation automatically and assigns it to the Root URL being analyzed.

The company's name is usually included in the HTML TITLE tag 600.However, the company's name could be included in other locations, aswould be known to those ordinarily skilled in the art within the purviewof the present specification. The layout of the address, if present,usually is in a standard recognizable format 602. Most businesses alsotend to include copyright notices near the bottom of their documents. Astring search for “copyright”, “&copy;”, and “&#169;” is performed nearthe bottom 604 of the home page. The company name usually appears nearthe copyright notice. A match of the organization or individual's namein the copyright field 420 and the TITLE field 425 provides the firstindication of the owner of the home page. If no match is found, then theURL is tagged for further analysis during the next iteration.

The next step is to analyze the URL for address 430 information.Addresses have an easily identifiable format. In the U.S., the format isthe city name followed by a comma and then followed by the full statename or abbreviation and finally a five or nine digit zip code. However,other common formats/methods also are possible and would be known tothose ordinarily skilled in this art field to locate the zip code. Thisstring is parsed in the HTML file after stripping all tags 435. The onlyinformation required is the 5-digit zip code since the city and statecan be determined by this field alone. YPLink stores addressesassociated with Root URLs and domain names in an address database 106.

If a phone format field is present then it is also extracted and stored440. U.S. phone field is a 10-digit field where the first three digitsrepresenting the area code are optionally enclosed in parentheses orseparated by a dash, space, or a period, and then followed by a 7-digitnumber which is separated by a dash, space, or a period after the thirddigit 445. Other similar methods of identifying a phone number are knownto those ordinarily skilled in the art.

The pair consisting of the company name and zip code are usually enoughto identify a business 455. A query is constructed using this pair andsent to a Yellow Pages database server. This database is indexed bybusiness names and zip codes. If a single match is found, then theresulting SIC code is assigned to the corresponding Root URL 460. Ifmultiple entries are matched, then the phone field is also included inthe query to assure that only a single entry is retrieved. If no matchis found, then the URL is tagged 465 for further analysis of lower-levelhyperlinks during the next iteration. The matching data is stored in anenhanced Yellow Pages database 108.

If no match is found at any level, then the page is tagged 450 as apersonal page with an SIC code assigned according to the closest matchbased on the Business Semantic Terminology database 110. This databaseis a proprietary thesauri of keywords relating business categories inthe Yellow Pages and other emerging industries such as Internettechnology to extended SIC codes.

While the invention has been described in terms of a single preferredembodiment, those skilled in the art will recognize that the inventioncan be practiced with modification within the spirit and scope of theappended claims.

For example, while the invention above has been described primarily interms of (e.g., implemented in) a software process and a systememploying software and hardware, the invention could also be implementedwith hardware as would be known by one of ordinary skill in the arttaking the present specification as a whole.

Having thus described my invention, what I claim as new and desire tosecure by Letters Patent is as follows:
 1. A method of classifying asource publishing a document on a portion of a network, comprising stepsof: electronically receiving a document; based on the document,determining a source which published the document; and assigning a codeto said document based on whether data associated with the documentpublished by the source matches with data contained in a database. 2.The method according to claim 1, wherein said portion of said networkcomprises a graphical multimedia portion of said network, said sourcecomprises a Web site publishing a home page, and said network comprisesthe Internet.
 3. The method according to claim 2, wherein said graphicalmultimedia portion of said network comprises the World-Wide Web (WWW)and said document comprises a Web document, wherein said step ofassigning a code includes determining that the Web site comprises afirst entity when there is a match of the Web site with said data, anddetermining that the Web site comprises a second entity when there is nomatch of the Web site with said data.
 4. The method according to claim1, wherein said step of determining a source includes: extracting adomain name from a predetermined uniform resources locator (URL)database; querying a database for storing registered domain names; andmerging an address database with predetermined data.
 5. The methodaccording to claim 4, wherein said predetermined data comprises YellowPages data, wherein said step of determining further comprises:characterizing uniform resource locators (URLs) by their InternetProtocol (IP) addresses including identifying a plurality of attributesbased on the IP addresses of new URLs, a new URL being retrieved andparsed into a domain name and directory path portions, and determining,based on said domain name, whether a selected URL is hosted on one of atrue server and a shared server.
 6. The method according to claim 5,said step of determining further comprising: for a shared server,determining a root path by searching for the given domain name in a newURL database and identifying common directory paths, wherein, when nomatch is present, the URL is processed subsequently at a lateriteration, and, when a match is present, the root path is set to amatching path.
 7. The method according to claim 6, wherein said step ofassigning a code comprises: automatically identifying a businessassociated with the source publishing said document, said business beinghosted on a Service Provider (SP) Web server.
 8. The method according toclaim 7, wherein said step of assigning a code further comprises:receiving a URL based on said determining step; and a URL determiningstep for determining whether said URL comprises one of a root URL and aleaf URL.
 9. The method according to claim 8, wherein said root URLcomprises an entry point for a home page on the World-Wide Web, and aleaf URL comprises a link below a root URL, wherein said URL determiningstep comprises: parsing said URL into a domain name component and adirectory path component; analyzing the domain name in said domain namecomponent to determine whether it is associated with an SP; when thedomain name is not associated with an SP, checking the directory pathcomponent to judge whether a directory path is missing, a missingdirectory path indicating a root URL; when the domain name is associatedwith an SP, checking whether a directory path does not exist to therebydetermine that said domain name comprises a root URL, and when adirectory path exists, then comparing the path to known SP ClientDirectory paths.
 10. The method according to claim 9, furthercomprising: when said URL is determined to be a root URL, analyzing ahome page associated with said root URL automatically to extract homepage data contained therein and assigning the home page data to the RootURL being analyzed.
 11. The method according to claim 10, furthercomprising: comparing said home page data with data in a predeterminedbusiness organizations database, wherein, when there is a match, saidcode is assigned to the corresponding root URL, and, when no match isfound, said URL is identified for subsequent analysis of lower-levelhyperlinks during a next iteration of said method.
 12. The methodaccording to claim 11, wherein when no match is found at any level, saidhome page is identified as a personal page.
 13. A method ofautomatically assigning a document a code for distinguishing afirst-type page from a second-type page, comprising steps of:electronically receiving a document; based on the document, determininga source which published the document; and assigning a code to saiddocument based on whether the source matches with data contained in adatabase.
 14. A search engine for use on a network for distinguishingbetween business web pages and personal web pages, comprising: means forparsing the content of a hyper-text markup language (HTML) at a webaddress and searching for criteria contained therein; means foranalyzing a uniform resources locator (URL) of the web address todetermine characteristics thereof of a web page at the web address;means for determining whether said criteria match with data contained ina database; and means for cross-referencing a match, determined by saiddetermining means, to a second database, to classify a source whichpublished the web page.
 15. A search engine according to claim 14,wherein said criteria include at least one of an address, a telephonenumbers, a facsimile number, a contact and a key-word contained in saidHTML, and wherein the characteristics of said web page include ageographical location and a web host computer.
 16. A search engineaccording to claim 14, wherein said database includes a BusinessSemantic Terminology database having information related to businesscategories in a Yellow Pages directory.
 17. A search engine according toclaim 14, wherein said second database includes a Yellow Pages database.18. A search engine according to claim 14, wherein said web pagecomprises hyperlinks, and said means for parsing comprises an indexerrobot for traversing said hyperlinks in said web page and a web indexdatabase, said indexer robot for indexing a content of said web pageinto said web index database.
 19. A search engine according to claim 14,wherein said means for analyzing comprises: means for determiningwhether said URL comprises one of a root URL and a leaf URL.
 20. Asearch engine according to claim 19, wherein said root URL comprises anentry point for the web page on the World-Wide Web, and a leaf URLcomprises a link below a root URL, said search engine furthercomprising: means for parsing said URL into a domain name component anda directory path component; means for analyzing the domain name in saiddomain name component to determine whether it is associated with an SP;means for checking the directory path component to judge whether adirectory path is missing, when the domain name is not associated with aservice provider (SP), a missing directory path indicating a root URL,and for checking whether a directory path does not exist to therebydetermine that said domain name comprises a root URL, when the domainname is associated with an SP; means for comparing the path to known SPClient Directory paths, when a directory path exists; means foranalyzing a home page associated with said root URL, when said URL isdetermined to be a root URL, thereby automatically to extract home pagedata contained therein; and means for assigning the home page data tothe Root URL being analyzed.